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1  Abstract 

Our  research  under  this  grant  has  developed  general  large-scale  global  optimization  tech¬ 
niques  in  the  context  of  solving  protein  structure  prediction  problems.  Over  the  course 
of  the  research  period  the  thrust  of  the  research  was  primarily  in  two  areas.  The  first 
area  provided  advances  in  our  techniques  for  handling  very  large,  complex  proteins  of 
arbitrary  secondary  and  tertiary  structure.  The  second  area  focused  on  advances  in  the 
efficiency  of  the  algorithm  on  large-scale  computing  platforms.  These  advancements  en¬ 
abled  us  to  significantly  improve  our  effort  in  the  CASP  community-wide  competition 
for  protein  structure  prediction.  During  the  CASP5  competition,  in  summer  of  2002,  we 
attempted  blind  predictions  of  20  proteins  with  sizes  and  structures  far  more  complex 
than  any  previously  attempted  by  our  group.  The  results  of  assessment  for  the  category 
of  proteins  which  contain  “new  folds”  (  i.e.  proteins  which  were  not  known  by  existing 
databases)  show  that  our  group’s  predictions  were  in  the  top  10%  of  groups  which  pre¬ 
dicted  these  targets.  Much  of  this  research  is  joint  work  with  the  bio-chemistry  group 
headed  by  Dr.  Teresa  Head-Gordon  at  UC  Berkeley,  and  the  computer  science  group  of 
Dr.  Silvia  Crivelli  at  Lawrence  Berkeley  Laboratory. 

2  Objectives  of  this  Research 

The  primary  objective  of  this  research  is  to' develop  large-scale  optimization  methods  for 
optimization  problems  that  arise  in  molecular  chemistry.  The  optimization  approaches 
are  being  developed,  however,  in  a  manner  that  makes  them  applicable  to  a  broad  class 
of  large-scale  optimization  problems.  The  main  optimization  problem  that  is  being  con¬ 
sidered  is  the  large-scale  global  optimization  problem.  This  is  the  key  problem  that  must 
be  solved  to  determine  the  configuration  of  a  molecule  or  macro-molecule  once  its  po¬ 
tential  energy  function  is  known.  Therefore  the  primary  objective  of  this  research  is  to 
develop  efficient  and  effective  large-scale  global  optimization  methods  for  determining  the 
structure  of  proteins.  This  includes  testing  of  the  methods  on  problems  that  constitute 
the  state-of-the-art  at  the  current  time  in  optimization  approaches  to  protein  folding. 
Since  the  solution  of  large  scale  unconstrained  optimization  constitutes  the  major  cost  of 
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the  global  optimization  calculations,  another  research  emphasis  is  finding  ways  to  reduce 
the  costs  of  the  local  optimizations  in  the  context  of  the  global  optimization  algorithm. 
In  addition,  due  to  the  size  and  difiiculty  of  the  problems  being  solved,  their  solution 
requires  the  use  of  very  powerful  computers,  generally  parallel  computers.  Therefore 
development  of  efficient  parallel  large-scale  global  optimization  methods  is  an  objective 
of  this  research. 

In  addition  to  the  global  optimization  research,  another  important  objective  of  this 
research  includes  the  development  of  interior  point  methods  for  for  very  large  equality 
and  inequality  constrained  optimization.  Both  of  these  topics  are  of  basic  importance  to 
the  Air  Force,  with  applications  ranging  from  construction  of  new  polymeric  materials  for 
the  global  optimization  work,  to  a  wide  range  of  design,  scheduling  and  control  problems 
such  as  aircraft  design  and  optimal  trajectory  problems  for  the  constrained  optimization 
research.The  final  thrust  of  this  research  has  been  in  the  area  of  large  scale  tensor  meth¬ 
ods  for  large-scale  systems  of  nonlinear  equations.  This  work  has  applications  in  the 
simulation  of  complex  physical  models  and  is  also  of  major  importance  to  the  Air  Force. 


3  Global  Optimization  Methods  for  Protein  Folding  Prob¬ 
lems 

Working  in  conjunction  with  biochemists  and  computer  scientists  at  U.C.  Berkeley  and 
Lawrence  Berkeley  Laboratory,  we  continue  to  be  one  of  the  leading  groups  that  is  taking 
an  optimization-centered  approach  to  one  of  the  leading  computational  challenges,  the 
solution  of  protein  folding  problems.  In  the  past  two  years,  we  have  expanded  and 
enriched  our  group’s  approach  so  that  it  is  now  able  to  deal  with  proteins  of  the  most 
general  complexity  (including  alpha  helices  and  beta  sheets)  and  of  as  large  a  size  as  is 
being  handled  computationally  (up  to  about  400  amino  acids).  Our  group  has  competed 
with  very  good  results  in  both  of  the  last  two  CASP  (Critical  Assessment  of  Techniques 
for  Protein  Structure  Prediction)  competitions,  in  summer/fall  2000  and  summer/fall 
2002. 

Our  main  research  accomplishment  has  been  the  critical  advancement  in  the  ability 
of  our  algorithm  to  predict  proteins  of  increasingly  complex  structure.  Through  the  de¬ 
velopment  and  testing  of  our  optimization-based  approach  for  incorporating  predicated 
beta-strand  and  beta-sheet  structure,  our  method  has  acheived  some  notable  compu¬ 
tational  success  in  predicting  the  structure  of  extremely  difficult  targets  in  the  recent 
CASP5  competition. 

In  previous  research  periods,  we  had  developed  a  stochastic/perturbation  approach 
for  solving  global  optimization  problems  from  molecular  chemistry.  The  key  to  this 
approach  is  a  step  that  performs  a  small-scale  global  optimization  with  only  a  small 
number  of  parameters  variable  and  the  remaining  parameters  temporarily  fixed,  followed 
by  a  large-scale  local  minimization  with  all  variables  varying. 

The  stochastic/perturbation  global  optimization  method  applied  to  protein  structure 
prediction  consists  of  two  phases.  The  first  phase  generates  a  set  of  good  initial  protein 
structures  using  information  specific  to  the  domain  of  protein  structure  prediction.  This 
information  is  the  prediction  of  secondary  structure,  which  is  readily  available  through 


servers  with  accuracies  averaging  around  75%.  Given  the  widespread  use  and  high  accu¬ 
racy  of  secondary  structure  prediction  methods,  it  would  be  impractical  not  to  attempt 
to  utilize  them  in  this  problem  domain  and  only  by  using  them  can  we  realistically  tackle 
reasonable-sized  problems. 

Secondary  structure  prediction  is  incorporated  into  the  first  phase  with  the  goal  that 
the  second  phase  begins  with  structures  that  have  much  of  the  predicted  secondary 
structure,  but  not  tertiary  strructure.  In  previous  research  periods,  this  was  accom¬ 
plished  solely  through  the  use  of  “biasing”  functions,  which  are  penalty  terms  that  bias 
the  structures  towards  predicted  secondary  structure,  used  within  the  context  of  local 
minimization.  Recently,  we  have  incorporated  a  powerful  visualization  tool,  developed 
by  our  colleagues  at  Berkeley  and  LBL,  to  use  in  conjuction  with  the  biasing  functions 
to  more  efficiently  create  starting  structures  with  predicted  secondary  structure.  This 
technique  was  especially  useful  in  creating  starting  structures  for  protein  targets  with 
extremely  complex  secondary  structures,  and  without  the  visualization  tool,  it  would  not 
have  been  possible  to  predict  structures  for  many  of  the  difficult  targets  we  attempted 
to  predict  in  CASP5. 

The  second  phase  of  the  algorithm  improves  upon  the  initial  structures  by  a  combina¬ 
tion  of  breadth  (work  on  a  variety  of  structures)  and  depth  (improve  the  current  lowest 
energy  structures).  It  does  this  by  a  combination  of  steps  that  change  the  structure  in 
a  restricted  manner  -  this  is  accomplished  by  small  scale  global  optimization  using  a 
stochastic  method  applied  to  a  selected  small  subset  of  the  parameters  -  and  local  mini¬ 
mizations  applied  to  the  full  structure.  A  key  to  this  step  is  selecting  a  small  subset  of 
parameters  that,  by  changing  their  value,  can  lead  to  a  substantially  improved  structure 
for  the  entire  problem.  A  list  is  kept  of  all  minima  obtained  from  the  local  minimizations, 
and  after  some  (fixed)  number  of  iterations,  this  list  is  clustered  via  pairwise  root  mean 
squared  deviation  (RMSD)  evaluations,  and  ordered  by  energy  value  within  each  cluster. 
The  RMSD  of  2  configurations  measures  the  closeness  of  their  structures,  i.e.  a  few  A 
difference  means  the  structures  are  very  similar,  and  beyond  15 A  usually  means  they  are 
not.  After  clustering,  if  the  stopping  criteria  have  not  been  met,  additional  iterations 
of  the  second  phase  are  applied,  starting  from  a  new  list  containing  the  lowest  energy 
minimizer  from  each  cluster. 

A  very  general  framework  that  shows  how  the  parts  of  the  global  optimization  ap¬ 
proach  fit  together  is  given  below  in  Algorithm  1. 

The  second  phase  of  the  stochastic/perturbation  algorithm  requires  many  compute 
cycles  to  perform  the  small-scale  global  optimizations  and  full-dimensional  local  opti¬ 
mizations.  During  the  period  of  this  research  grant >  we  have  successfully  adapted  our 
algorithm  to  several  massively  parallel  computing  environments,  including  the  Cray  T3E 
at  the  Pittsburgh  computing  center  and  the  IBM  SP  RS/6000  at  NERSC.  Computational 
requirements  for  the  exremely  large  proteins  tackled  in  CASP5  required  the  use  of  from 
64  to  256  prosessors  to  complete  the  predictions  within  the  submission  deadlines  for  the 
competition. 

Our  use  of  smoothing  within  our  global  optimization  algorithm  has  has  been  an  im¬ 
portant  component  over  the  past  several  years,  and  we  present  our  current  experimental 
findings  here.  By  smoothing  we  mean  replacing  the  objective  (energy)  function  with 
a  smoother  version  that  is  intended  to  retain  the  coarse  grain  structure  of  the  original 
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Algorithm  1:  Stochastic/Perturbation  Global  Optimization  Algorithm  for  Protein 

Structure  Prediction 

1.  Phase  1:  Coarse  Identification  of  Configurations  : 

Generate  initial  configurations  containing  domain  specific  tendency  information. 

(a)  Generate  configurations  with  (server  obtained)  predicted  secondary  structure 
using  the  visualization  tool 

(b)  ’’Biased”  Full-Dimensional  Local  Minimizations  : 

Use  server  obtained  information  to  create  biasing  terms  for  predicted  secondary  structure. 
Perform  a  local  minimization  from  a  subset  of  the  sample  points,  using  “biasing”  penalty 
functions  to  superimpose  predicted  secondary  structure  on  the  standard  energy  surface. 

(c)  “Unbiased”  Full-Dimensional  Local  Minimizations  : 

Perform  a  local  minimization  from  each  “biased  minimizer”  using  the  “unbiased”  energy 
function.  Save  these  local  minimizers  for  Step  2a. 

2.  Phase  2:  Improvement  of  Local  Minimizers: 

For  some  number  of  iterations: 

(a)  Select  a  Configuration  and  Small  Subset  of  Parameters  to  Improve: 

From  the  list  of  full-dimensional  local  minimizers,  select  a  local  minimizer  to  improve.  Then 
select  a  subset  of  the  parameters  of  this  configuration  to  optimize  in  step  2b. 

(b)  Small  Sub-problem  Global  Optimization: 

Apply  a  global  optimization  algorithm  to  the  energy  of  the  selected  configuration  with  only 
the  selected  parameters  as  variables. 

(c)  Full-Dimensional  Local  Minimization  : 

Apply  a  local  minimization  procedure,  with  all  the  parameters  as  variables,  to  the  lowest 
energy  configurations  that  resulted  from  step  2b,  and  merge  the  new  local  minimizers  into 
the  list  of  local  minimizers. 

(d)  Cluster  Local  Mimima  and  Test  for  Convergence 

Cluster  the  list  of  local  minimizers  via  pairwise  RMSD  and  if  the  stopping  criteria  has  not 
been  met,  repeat  all  steps  of  Phase  2  from  a  new  list  of  local  minimizers  containing  the 
lowest  energy  minimizer  from  each  cluster. 


function  but  to  damp  the  finer  grain  structure.  The  smoothed  function  should  thus  be 
easier  to  locate  low  minimizers  on,  while  still  serving  as  a  guide  to  the  structure  of  the 
original  energy  function.  Previous  to  this  research  grant  we  had  developed  novel  analytic 
techniques  for  smoothing  molecular  chemistry  objective  functions.  They  rely  on  replac¬ 
ing  the  Lennard-Jones  and  electrostatic  terms  with  parameterized,  smoother  variants. 
Our  research  confirmed  that  our  approaches  definitely  reduces  the  number  of  minimizers 
substantially.  It  also  revealed  the  phenomenon  of ’’order  flips”,  that  the  lowest  minimizer 
of  the  smoothed  function  may  not  track  back  to  the  lowest  minimizer  of  the  unsmoothed 
function.  This  indicated  that  global  optimization  techniques  remain  crucial  even  with 
smoothing. 

Results  on  simple  proteins,  polyalanine  and  metenkephalin,  showed  that  the  use  of 
smoothing  significantly  enhances  the  efficiency  of  our  global  optimization  approach.  Our 
more  recent  experimental  work,  particularly  on  some  of  the  CASP5  targets  subsequent 
to  the  competition,  has  once  again  shown  the  utility  of  using  smoothing.  The  success  of 
smoothing  compared  to  no  smoothing  is  particularly  evident  on  alpha-helical  proteins, 
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but  not  as  obvious  with  beta  proteins.  We  see  good  potential  for  improving  this  exper¬ 
imental  work  through  incorporation  of  smoothing,  and  intend  to  generalize  the  success 
on  helical  proteins  to  all  protein  structural  types. 

Some  of  the  our  recent  successes  in  protein  structure  prediction  include:  (1.)  During 
the  CASP4  (summer/fall  2000)  competition,  we  submitted  the  best  prediction  of  any  of 
the  roughly  150  groups  on  one  of  the  large  alpha-helical  proteins,  and  our  submissions 
were  among  the  best  predictions  on  all  of  the  several  ”new  folds”  that  we  attempted. 
New  folds,  where  the  structure  does  not  closely  match  the  structure  of  known  proteins, 
are  the  hardest  problems  and  the  ones  for  which  optimization-centered  approaches  are 
primarily  intended.  (2.)  In  the  CASP5  (summer/fall  2002)  competition,  our  submissions 
were  among  the  best  predictions  on  the  ten  ”new  fold”  targets  that  we  attempted.  Our 
predictions  on  these  targets  ranged  from  the  70th  to  the  98th  percentile  among  all  the 
165  groups  in  the  competition.  Overall  we  placed  about  15th  of  the  165  groups  in  the 
assessments  of  the  new  fold  category;  most  of  these  groups  are  using  far  more  chemical 
knowledge  that  our  more  purely  optimization-centered  approach.  We  appear  to  be  one 
of  the  two  leading  groups  (with  roughly  equal  performance)  whose  protein  structure 
predictions  are  based  primarily  on  energy  optimization. 


4  Algorithms  and  Software  for  Large  Scale  Constrained 
Optimization 

As  part  of  this  grant  we  have  been  deeply  involved  in  development  of  methods  for  large 
scale  nonlinear  optimization.  We  have  been  active  in  this  area  on  three  fronts:  en¬ 
hancement  of  the  interior  point  algorithm  KNITRO,  a  mathematical  investigation  into 
failure  of  line-search  algorithms  for  such  problems,  and  beginning  development  of  a  new 
constrained  optimization  algorithm  based  on  successive  linear  programming. 

The  algorithm  KNITRO  (formerly  called  NITRO)  was  developed  by  us  under  a  pre¬ 
vious  AFOSR  grant,  in  collaboration  with  Jorge  Nocedal  at  Northwestern  University. 
It  uses  trust  regions  in  conjunction  with  an  interior  point  strategy,  to  deal  with  large 
numbers  of  inequality  constraints,  by  approaching  the  optimum  from  the  interior  of  the 
feasible  region.  Over  the  last  couple  of  years  KNITRO  has  developed  into  one  of  the 
best  performing  codes  for  large-scale  constrained  optimization  problems,  together  with 
LOQO,  developed  at  Princeton  and  SNOPT,  developed  at  Stanford  and  UCSD.  This  as¬ 
sessment  is  borne  out  in  benchmarking  performed  by  Hans  Mittelmann  at  Arizona  State 
University.  In  the  period  2000-2003  we  have  concentrated  on  making  KNITRO  into  a 
useful  and  available  piece  of  software,  and  on  some  crucial  extensions  to  its  capabili¬ 
ties.  In  addition  we  completed  a  paper  analyzing  the  convergence  of  the  algorithm  which 
appeared  in  Mathematical  Programming. 

In  the  last  two  years,  KNITRO  has  been  made  available  on  the  NEOS  server.  (NEOS 
is  an  on-line  facility  maintained  by  Argonne  National  Laboratory,  that  is  available  to  the 
general  public  and  solves  over  10,000  user-submitted  optimization  problems  a  month.) 
In  the  first  4  months  of  this  year  NEOS  users  have  accessed  KNITRO  over  2700  times.  It 
has  been  used  to  solve  problems  in  many  applications  areas.  A  group  of  us  has  formed  a 
company,  Ziena  Optimization,  which  is  marketing  the  most  powerful  version  of  KNITRO 
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to  companies,  and  which  has  received  an  SBIR ^grant ,  .  vements  to  this  algorithm. 

In  the  last  two  years  second  derivatives  to  approximate  the  La- 

The  original  version  of  KNITRO  used  ex  ^  ^  ^  approach  is  impractical 

grangian.  We  did  have  havaaq  £fd  Now  we  have  developed  a  finite-difference 

for  problems  with  many  degrees  of  fre  •  established  approach  uses  gradient  m- 
truncated  Newton  version  of  the  algorit  _  •  iteration  to  develop  an  implicit 

formation  gathered  in  the  course  of  a  This  allows 

approximation  to  the  Hessian.  It  ts  na  ura  y  i  ^  without  constructing 

un  to  obtain  fast  convergence  "'"•?„  w te  il^eint  was  to  add  a  facility 
to'detec"  "hi InconTt ent  or  unsalable  constrain,,  and  terminate 

at  every  iteration  in  the  interest  of  efflcrence  and  flex*  l.ty, thereare  PP 

:"aS:,"rfd^d  om—e  in  a  paper  that  has  appeared  in  the 

journal  Computational  Optimization, sludy  of  failures  of 
We  have  also  m  the  course  o  th“  r“e““TR0  0ne  of  the  features  of  KNITRO  rs 
line  search  methods,  which  is  relevant  t  control  the  step.  The  advantage  of 

its  use  of  a  trust  region  rather  than  a ^  sear^to  In  order  to 

this  is  that  we  can  obtain  a  more  ro  g  ^  the  kinds  0f  problem  where  a  line 

assess  the  importance  of‘h,  >ssue  we  ma  ^  ^  Wc  were  able  to  establish  that 

search  method  would  fail  and  a  trust  g  deficiency  of  order  one  at  a  point, 

whenever  the  constraint  derivative matt*  h»  a  rm* defloen  ^  ^  ^  ^  This 

there  is  about  an  even  chance  that  a  to  se  nonUnear  partial  differential 

is  an  important  issue,  there  are  know  P  and  t  dy  indicates  that  it  is 

equations  where  line  search  methods  fail  in  practice  ano  ^  As  ^  fti6 

worthwhile  to  try  to  extend  trust  region  “e  wete  much  less  pkely  to  occur  in 

study,  we  were  also  able  to  *°w  a  ™  t  fol  the  similar  performance  of  trust 

— S — hods  Jn  such  problems.  A  paper  describing  this  work 

will  appear  in  Mathematical  Programming  in ■  ear ^y  involves  beginning  develop- 

A  final  aspect  of  our  constrained  op  constrained  optimization,  that  should 

ment  of  a  quite  different  approach  °  pI0blems.  A  weakness  of  interior  point 

have  crucial  advantages  in  an  important  class  of  problems  is  tar  from 

methods  like  KNITRO  is  that  they  Methods lend  to  ml  poor  use  of  a 

"^t“"  sTo"ch  a -warm  start-  is  often  available,  for  example, 

active  set  method  using  repeated  J^feasMe  region,  and  are  able  to  make 

active  set  methods,  most  of  which  use 


successive  quadratic  programming,  are  limited  in  the  size  of  problems  they  can  solve, 
especially  when  the  number  of  degrees  of  freedom  is  large.  To  be  able  to  handle  larger 
problems  we  have  turned  to  successive  linear  programming  as  a  means  of  estimating 
the  correct  active  set.  We  then  solve  an  equality  constrained  quadratic  minimization 
problem,  using  this  active  set.  This  approach  is  related  to  an  earlier  method  due  to 
Roger  Fletcher,  but  makes  more  use  of  trust  regions,  and  is  better  able  to  handle  negative 
curvature.  Preliminary  experiments  with  such  a  method  show  it  to  be  efficient  and  quite 
robust  compared  to  other  methods.  A  paper  describing  this  method  has  been  accepted  by 
Mathematical  Programming  Series  B.  Currently  we  are  working  on  making  this  method 
more  efficient  by  using  inexact  solutions  of  the  linear  programs. 


5  Tensor  Methods  for  Large  Scale  Systems  of  Nonlinear 
Equations 

The  main  objective  of  this  research  was  to  combine  multiple  approaches  for  solving  sys¬ 
tems  of  nonlinear  equations,  primarily  tensor  methods  and  Krylov  subspace  methods,  to 
develop  an  effective  large-scale  nonlinear  equations  solver.  We  developed  three  Krylov- 
based  methods  for  iteratively  solving  the  local  tensor  model  and  embedded  them  within 
an  inexact  framework  for  solving  large,  sparse  nonlinear  equations  problems.  Over  the 
course  of  this  research,  we  made  several  contributions,  and  we  summarize  the  four  prin¬ 
cipal  ones  here. 

First,  we  investigated  the  performance  of  direct  tensor  methods  on  smaller  problems 
over  a  range  of  conditionings,  from  well-conditioned  to  ill-conditioned  to  singular.  Prior 
to  this  investigation,  studies  on  direct  tensor  methods  only  focused  on  singular  problems 
or  on  general  problems  that  are  well-conditioned,  and  had  not  investigated  the  perfor¬ 
mance  of  tensor  methods  on  ill-conditioned  problems.  The  results  showed  that  direct  ten¬ 
sor  methods  outperform  Newton’s  method  as  the  problems  become  more  ill-conditioned. 
Specifically,  the  results  show  that,  eventual  quadratic  convergence  notwithstanding,  the 
performance  of  Newton’s  method  degrades  as  ill-conditioning  grows,  whereas  tensor 
methods  are  relatively  unaffected  to  only  mildly  affected  (for  larger  rank  deficiencies). 
This  observation  has  particular  relevance  for  practical  problems  where  the  Jacobian  at 
the  solution  is  very  ill-conditioned,  such  as  in  the  identification  of  bifurcation  points  or 
in  path  following  problems  that  try  to  locate  turning  points. 

Second,  we  developed  a  new  robust  global  strategy  for  tensor  methods,  called  the 
curvilinear  linesearch.  This  linesearch  combines  the  Newton  and  tensor  steps  into  a  sin¬ 
gle  parametric  step,  which  guarantees  a  monotonic  decrease  on  the  tensor  model  and 
also  asymptotically  approaches  the  Newton  direction  as  the  step  length  shrinks  to  zero, 
guaranteeing  descent  on  the  nonlinear  equations.  These  theoretical  properties  make  the 
curvilinear  linesearch  more  attractive  than  other  ad  hoc  linesearches  that  use  the  ten¬ 
sor  step  and  Newton  direction  separately,  such  as  the  standard  tensor  and  TENSOLVE 
linesearch.  Our  tests  showed  that  the  curvilinear  linesearch  outperforms  the  standard 
tensor  and  TENSOLVE  linesearches,  both  on  a  small-scale  test  set  and  on  large-scale 
problems.  The  curvilinear  linesearch  tensor  method  also  appears  to  be  more  robust,  in 
terms  of  success  versus  failure  in  solving  problems,  than  both  linesearch-based  Newton’s 
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method  and  the  standard  tensor  robust  per-  S 

« •  "°rthy  imp—  over 

model  using  Krylov  subspace  technique concepts  but  are  distinctly  different 
The  three  methods  share  the  use  of  b  features  of  these  new  methods 

to  heir  approach  to  solving  the  ^7sLnZ,%  Jacobian-vector  product 
L  that  their  costs  are  stall* :  to  fU^Z^Jons  beyond  GMRES  per  solve, 
at  each  iteration  and  O(nm)  addi  iona  ^  ^  within  a  specified  tolerance,  ma  mg 

a  projected  tensor  term  and  computes  a  ensor 
step  with  an  unpredictable  residual  error  ^  solve  the  local  tensor  model 

The  new  tensor-Krylov  methods  prop  large.SCale  tensor  methods  (the  metho 

in  a  novel  fashion.  Compared  to  the  oth  |ulliam  and  Feng),  they  are  the  only 

of  Bouaricha,  and  the  tensor-GMRES  met  tensor  step  from  the  unprojected 

large-scale  tensor  methods  that  produ  residual  of  the  exact  tensor  step  to  withi 

Sr  model  and  that  approach  the — solves  the  unprojec«ed  = 
a  specified  accuracy.  For  instance,  ,  £  tbe  tensor  model  and  the  GM 

model,  but  the  relationship  between  ®  *  Also,  the  Feng-Pulliam  method  solves 

tolprance  used  in  the  two  linear  solves  is  not  K  h  projection,  and  again  lacks 

"" model, -hlch“e^mat.on.nthapJdttai  ^  ^ 

.  direct  relationship  betvmen  the  GMRE  ^  deyeloped  £or  Newton-Krylov 

Krylov  methods  also  can  util  restarting 

methods,  including  preconditiomng  and^esta^^  ^  ^  solve. r  ftrnne- 

Last,  we  incorporated  these  three  curvilinear  linesearch,  for  different 

w0*  with  various  linesearch ^options  mcluing  th  ^  ^  Md  TK2+.  We 

^  w-pSt  -- — -  *  - o£  ^  " 

Pt0brng  an  of  the  methods,  the  new  these 

method  by  Feng  and  Pulliam  axe  th* ^mos  Newton.GMRBS,  particularly 

large-scale  tensor  methods  hold  advan  g  have  yen  t0  be  more  robust  than 

conditioned  problems.  when  the  tensor-Krylov  methods  employ 

Newton-GMRES  on  most  problems,  especia  y 

the  curvilinear  linesearch.  sufficient  for  a  good  step,  which 

For  some  problems,  the  proje^d ^  a  more  efflc,ent  local  solver  wrth 
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most  always  performs  the  best  on  the  easier  problems,  but  it  performs  poorly  on  the  more 
difficult  problems,  where  the  tensor-Krylov  methods  axe  more  efficient.  Among  just  the 
new  tensor-Krylov  methods,  it  appears  that  TK2  and  TK2+  edge  out  TK3  in  terms  of 
efficiency,  particularly  when  using  restaxts.  There  is  usually  little  difference  between  TK2 
and  TK2+,  so  the  additional  complexity  of  TK2+  may  limit  its  usefulness  in  practice. 

Thus,  the  new  tensor-Krylov  methods  are  especially  aimed  at  large-scale  problems 
that  are  highly  ill-conditioned  or  singular,  where  algorithms  based  on  Newton’s  method 
exhibit  very  slow  convergence  and  where  tensor  methods  excel.  Of  particular  interest 
are  PDE  problems,  which  are  the  focus  of  many  large-scale  algorithms.  For  example, 
multigrid  methods  may  solve  a  variety  of  PDE  problems  efficiently,  but  the  coarse  grid 
solution  can  be  difficult  to  obtain  in  some  cases,  which  could  be  a  worthy  application 
of  tensor-Krylov  methods.  Our  results  suggest  that  the  tensor-Krylov  methods  are,  on 
average,  more  efficient  and  robust  than  Newton-GMRES.  In  addition,  our  results  show 
that  tensor-GMRES  is  a  viable  and  competitive  algorithm  that  should  be  considered. 
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